Luigi HDFS

This function defines a Luigi task that reads a CSV file from HDFS, calculates a random column, and writes the result back to HDFS.

Luigi mission 2024-12-16 12:04:20 4 views

hdfs google-cloud-storage

This function copies a file from Hadoop Distributed File System (HDFS) to Google Cloud Storage (GCS).

The type of code 2024-12-16 11:47:34 3 views

Luigi HDFS

This function generates a random file path under a specified directory on HDFS. By specifying the directory as a parameter, the function returns a path that includes a randomly generated file name.

The type of code 2024-12-16 11:42:23 4 views

Luigi Postgres

This code defines a function named random_target that randomly generates a Luigi target for either Postgres or HDFS. Another function, process_data, processes data based on the type of target. If the target is Postgres, it executes a randomly generated SQL query and prints the results. If the target is HDFS, it reads the file and prints each line.

Function 2024-12-07 16:28:00 7 views

The packages

This code defines a function named xxx that accepts a date as input, generates random data, and stores the data in HDFS and PostgreSQL databases.

Function 2024-12-07 16:17:35 3 views

Luigi HdfsTarget

This code defines a function named random_task that randomly selects a sub-module and function from Luigi and returns a custom Luigi task function. This custom function accepts input and output files as parameters and uses the randomly selected sub-module and function to process data.

Luigi Task Function 2024-12-07 16:11:19 3 views

Luigi Parameter

This code defines three tasks based on the Luigi library, each for different types of tasks: file output with a parameter, data transfer from S3 to HDFS, and querying data from a PostgreSQL database.

Luigi Task 2024-12-07 15:53:43 3 views

Luigi HDFS

This code defines a Luigi task for fetching data from HDFS (Hadoop Distributed File System). It uses basic functionalities of the Luigi library to create a task, including input/output definitions and runtime logic.

Luigi Task 2024-12-07 15:49:12 4 views

Luigi HDFS

This function uses the Luigi library to create a simple task that randomly selects one of two arguments and outputs it to HDFS. This demonstrates how to use Luigi for task scheduling and data operations.

Function 2024-11-30 16:25:14 4 views

Luigi HDFS

This function defines a data processing task using the Luigi framework. The task reads a CSV file from HDFS, computes the square of each data item, and saves the result to a new CSV file.

Data processing 2024-11-30 16:06:09 3 views